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■ VISION & 

LEADERSHIP 

Assessing Students 
with Disabilities: 

Moving Assessment Forward 
through Universal Design 



Ted S. Hasselbring 
Preston Lewis 
Margaret E. Bausch 



he amendments to the Individuals with Disabilities Education Act 
(IDEA) of 1997 required all states to include students with disabilities in their 
state assessment and accountability systems. Compliance with this federal 
mandate made it necessary for states to develop accommodations to allow 
students with disabilities to participate in state assessments. Because most state 
assessments are administered in a paper format, students need reading, process- 
ing, organizational, and other skills to participate in the assessment. Some 
students depend on accommodations, such as extended time, a specialized 
setting, magnified text, or a human reader who reads text, as well as special 
paper or booklets. While such adaptations have done much to support inclusion 
of students with disabilities, there are several disadvantages. Students often 
regard the accommodations as stigmatizing, and fewer students use these 
supports as they get older.' Furthermore, traditional accommodations, such as 
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read- aloud support, are not always administered to students individually. They may 
be administered in a group setting, where students are less inclined to ask for 
support, such as having passages reread, a strategy available to and often used by 
nondisabled students. 

In the remainder of this article, we will present an innovative approach to 
standardized testing that allows students access to assessment materials and reduces 
the stigma associated with most testing accommodations. We will explain how this 
method utilizes the principles of universal design for learning and capitalizes on the 
flexibility of online presentation of content to better meet the needs of students 
participating in high-stakes achievement tests. The online assessment procedures 
were developed under the direction of the Kentucky State Department of Education 
(KDE) in collaboration with one of the authors, Preston Eewis, formerly of KDE 
Exceptional Children Services. The other two authors served as advisors during the 
pilot projects. 

Universal Design for Learning 

The 2004 reauthorization of IDEA attempted to level the playing field for 
students with disabilities by requiring states to develop universally designed assess- 
ments. The law states that “the state educational agency (or, in the case of a 
districtwide assessment, the local educational agency) shall, to the extent feasible, 
use universal design principles in developing and administering any assessments 
under this paragraph.”^ The principles of universal design for learning (UDL) grew 
out of the field of architecture. In the 1970s, a growing number of architects began 
to develop buildings that did not require retrofitting to accommodate all individu- 
als, especially those with disabilities. The method of designing buildings that 
accommodated the access and movement of all individuals was called universal 
deagn. One unexpected result of universally designed buildings was that all indi- 
viduals benefited from this type of construction. The ease of access and movement 
through universally designed buildings was appreciated as much by individuals 
pushing strollers as it was by persons in wheelchairs. 

Beginning in the 1980s, the education community embraced the principles of 
universal design and applied them to teaching and learning. One of the first groups 
to advocate for the use of UDL was the Center for Applied Special Technology 
(CAST). David Rose and Ann Meyer, both from CAST, suggested “a set of prin- 
ciples that forms a practical framework for using technology to maximize learning 
opportunities for every student.”^ UDL focuses on designing instruction that is 
accessible from the outset, rather than retrofitting the learning materials with a 
variety of accommodations. In its purest form, UDL instructional materials and 
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instruction are designed to meet the learning needs of individuals with varying levels 
of ability. 

Three guiding principles to universally designed materials have been suggested 
by Rose and Meyer. They include (1) multiple representations of content, (2) 
multiple means of expression and control, and (3) multiple options for engagement. 
Each of these principles offers the learner various and increased forms of control and 
engagement with the materials. 

M ultiple means of presentation. Materials that offer a wide range of media 
and formats give the learner increased options for access. For example, the tradi- 
tional textbook creates a number of barriers to various groups of the population. For 
those who have difficulty with decoding words or comprehending text, the tradi- 
tional textbook creates an enormous barrier to learning. For students with severe 
physical problems, the pages of a book may be too difficult to turn. For some 
students with visual impairments, a traditional textbook is useless for trying to access 
information. And for all students, a backpack full of textbooks can pose a daunting 
and physically demanding task. 

A universally designed text would offer multiple ways to represent the content. 
For example, a digital text offers advantages over the traditional form for many 
students with disabilities. Using a computer with digital text, students with reading 
or vision problems can have the text read to them. The font size, type, and color can 
be controlled by the learner. Multiple digital pictures and movies can be used to 
supplement the text, tfyperlinks can be used to guide the learner to other informa- 
tion relevant to the topic being studied or provide assistance with definitions of 
words. 

M ultiple means of expression. Materials developed with universal design give 
students multiple ways to demonstrate mastery or understanding of what they have 
learned. For example, a universally designed writing environment could provide 
students with writing support, such as word prediction, read-back of what they have 
written, or dictionary and thesaurus support, all in the same environment. These 
features can help students with reading and writing difficulty, visually impaired 
students, and even second-language learners demonstrate knowledge, mastery, and 
understanding. 

Text is not the only way students can express what they have accomplished. 
Other forms of expression include multimedia presentations, video projects, audio 
recordings, and even dramatic or musical performances. With UDF, students are 
given the opportunity to demonstrate knowledge in alternative ways. 
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M ultiple options for engagement. A basic principle of UDL is that students 
are given materials that can be accessed in a variety of formats to enhance engage- 
ment. For example, digital materials might be accessed in visual or auditory formats. 
Students have the option of wearing earphones when listening to text to reduce 
outside distractions. The ability to control the speed at which text is read allows 
students to select a personally optimal speed. The student controls whether the text 
is reread and how many times it is reread. This form of control often promotes an 
enhanced level of engagement. 

UDL and the Assessment of Students 

Test accommodations for students with disabilities have been commonplace for 
a number of years. However, the design of assessments using UDL principles is 
relatively new, especially for high-stakes state tests. Like UDL, universally designed 
assessments should be constructed from the beginning to allow all students optimal 
opportunity to participate in the assessment and to achieve a score that represents 
what they know and understand. 

Our experiences with representatives from state departments of education at 
national conferences and other meetings indicate that increasing consideration is 
being given to universal design in the delivery of state assessments. Yet, only a few 
states (e.g., Kentucky and Kansas) have actually moved toward incorporating UDL 
into their accountability testing systems. Many states have found that they do not 
have the technology infrastructure necessary to provide universally designed assess- 
ments statewide. Although the use of technology is not a requirement with respect to 
universal design, the dynamic and flexible nature of technology does lend itself to 
the principles and delivery of a universally designed assessment. 

Kentucky’s Universal Design for Learning Initiative 

Perhaps one of the best examples of the benefits of including UDL in the 
design and delivery of a state assessment comes from recent experiences in Kentucky. 
It is important to note that Kentucky’s attempts to provide a universally designed 
assessment originated out of a larger state effort to make UDL a statewide instruc- 
tional initiative. The Kentucky UDL initiative consists of three components: (1) 
computerized literacy supports, (2) distribution of accessible digital curriculum, and 
(3) accessible electronic state assessment. Each of these components is explained in 
detail below. 

Computerized literacy supports. The major factor that limits disabled stu- 
dents’ access to the general curriculum is their inability to read at a level comparable 
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to that of their same-age peers who do not have disabilities.^ It was estimated by the 
Council for Exceptional Children that up to 80 percent of students with learning 
disabilities experience problems with reading.^ Reading problems are exacerbated by 
the absence of alternatives to the print medium in classrooms. The lack of alterna- 
tives poses a major barrier to access to the general curriculum that directly affects 
student performance and participation in state assessment. 

The inaccessibility of print-based materials is a prime example of where 
universal design could benefit students, both those with and without disabilities. It is 
too often assumed that if students do not have the skills or ability to access print, 
they cannot understand what is being taught. However, after UDL is implemented, 
teachers have discovered that students can understand the content of instruction 
when the content is presented in alternate formats. One key way to overcome the 
print barrier is to use media and tools that are available through digital technologies. 
One such technology-based method, computerized reading supports, has been 
shown effective in helping students with disabilities obtain access to printed mate- 
rial.^ With this in mind, in 2000, the Kentucky Department of Education (KDE) 
began to address the issue of using technology to enhance access to the general 
curriculum by attempting to provide text-to-speech technology (i.e., a text reader) in 
every school in the Commonwealth. 

After examination of the various text reader tools available, the KDE negoti- 
ated an agreement with TextHELP, Ltd. for a volume purchase of 100 to 200 site 
licenses per year. As a result, approximately 1,300, or 95 percent, of Kentucky 
schools have acquired site licenses, mostly for Read & Write Gold software. Read & 
Write Gold provides literacy support to students through functions such as speech 
feedback, spell checking, word prediction, and a talking dictionary. 

There were two main reasons that the TextHELP product was chosen over 
other popular commercial tools. The first was cost. The KDE realized that the 
purchase of single-station copies of a text reader for selected computers in a school 
would not provide students the access needed across educational settings. Students 
needed this technology in special education resource rooms at isolated workstations 
but even more so in regular education settings, where the greatest barriers to the 
general curriculum tend to occur. TextHELP offered the most affordable price for a 
site license version of a text reader. Distribution of the site licenses also included 
assistance with teacher training and technical support. 

The second reason for selection of the TextHELP product was the inclusive 
nature of the tool’s browser. The Read & Write tool bar appears and operates directly 
from any window the user has open, whether the window contains a word process- 
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ing document, an e-mail message, or a Web page. In contrast, all other similar 
commercial products require text to be imported into the software before it is read to 
the user. Not only does this process create an additional step for students, it also 
poses a security issue when used with state online assessments because questions used 
year after year could no longer be kept confidential once they were copied to a local 
computer. 

Other literacy tools of the Read & Write product are equally important, such 
as the speaking spell check, dictionary, and word prediction tools. For instance, the 
word prediction tool has been very valuable in helping students develop their writing 
portfolios. This is true both for students with disabilities and for other students. 
These students also benefit from the vocabulary resources. In addition, with this 
product, students can listen to their compositions, reflect on what they have written, 
and make desired corrections. 

D istribution of accessible digital curriculum. The use of computerized literacy 
supports to access the general curriculum is predicated on having instructional 
content and assessment accessible in a digital format. The KDE effort to meet this 
challenge was facilitated by passage of state legislation (Senate Bill 243) in 2002 that 
gave “preferential procurement” incentives to publishers who agreed to give KDE 
their text material in accessible format on CDs.^ Annual provision of the digital text 
is tied to the KDE textbook adoption cycle, and about 80 percent of the materials 
are being provided each year from publishers for distribution to eligible students 
with disabilities. More than 1,000 texts are now available. KDE has collaborated 
with the University of Louisville to provide a repository for receipt, review, and 
distribution of the material through the Kentucky Accessible Materials Consortium 
(KAMC at http://kamc.louisville.edu/kyecontent/). Local schools can view and 
request digital text material through the Kentucky Accessible Materials Database 
(KAMD at http://apps.kde.state.ky.us/kamd/) located at the KDE. 

A major issue that had to be addressed in the creation of this state digital text 
repository is copyright protection. This required developing a process for electroni- 
cally embedding code numbers in every individual CD sent for student use. The 
numbers discourage copyright infringement by creating a tracking mechanism for 
unauthorized releases. Each participating school was also required to appoint a 
person to serve as the digital rights manager (DRM) to oversee appropriate distribu- 
tion and use of the CDs by authorized teachers and students with disabilities. 

While the KAMC provides an extensive and unique resource to schools for 
accessing digital text, local general curriculum content comprises an array of material 
that goes beyond use of specific textbooks. Providing access to the complete local 
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general curriculum requires massive district and school commitment to using other 
methods, such as Web searches and scanning, to procure the total curriculum in 
accessible format. The KDE has facilitated this effort, in part, by providing 500 
scanners to selected local schools that use the Read & Write software. The subse- 
quent challenge has been for schools and/or districts to develop a local distribution 
system whereby teachers and students can access this electronic content in a way that 
is efficient but that still includes safeguards and usage limitations to assure copyright 
is preserved and protected. 



Accessible electronic state assessment; C AT S 0 niine. With computerized 
reading supports and digital text in use across the Commonwealth to access the 
general curriculum, the KDE soon recognized the need to develop and deliver the 
state assessment in accessible electronic format. Conversations began in the late 
1990s between the KDE and their assessment vendor, CTB McCraw ffill, about 



providing technology-based accommodations for eligible 
students with disabilities for the Kentucky Core Content Test 
(KCCT). At that point, additional expertise was needed for the 
digital test conversion and delivery. This led to including experts 
at eCollege of Denver in the discussions. eCollege had adminis- 
tered the Kentucky Virtual High School (KYHS), so its back- 
ground in electronic design and accessibility was well estab- 
lished. As a result, a pilot online assessment project was 
launched in the spring of 2002, with 21 schools and 61 stu- 
dents, to test and refine the online interface. Another pilot 
study was conducted in the fall of 2002, with 74 schools and 
457 students. This study included content across the seven 
grade levels of the KCCT, using previously released test items. 
This pilot also helped to identify local school and student access 
issues that had to be resolved for successful implementation. 

The positive responses from teachers and students taking 
part in the two pilot studies^ '® brought about Kentucky’s first 
electronic assessment in the spring of 2003, known as CATS 
(Commonwealth Accountability Testing System) Online. 



The eligibility criteria for use of 
the CATS Online 
accommodation require that 
students 

• have an lEP, 504 Plan, 
or Program Service Plan 
(PSP) that specifies the need 
for a reader or assistive 
technology as an 
instructional and assessment 
accommodation 

• routinely (daily to weekly) 
use text reader, screenreader 
(text to speech), or related 
technology to access printed 
material in classroom 
instruction and assessment 

• access and use the CATS 
Online Practice Area until 
they are able to navigate the 
site and independently read 
the content with their text 
readers, screenreaders, or 
related technology 



Student and teacher comments throughout this paper are from these two pilot 



studies. 



Twenty-nine schools and 204 students with disabilities participated in the 
2003 online administration. In 2004, more than twice this number took the test 
online, with 74 schools and 510 students using the electronic accommodation. The 
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most recent administration in spring of 2005 included 147 
schools and 1,230 students with disabilities, plus 35 
students with limited English proficiency (LEP). (See 
sidebar for eligibility criteria.) 

Of particular importance is the use of the Practice 
Area of the assessment, which includes released items from 
all content and KCCT grade levels. Before students can 
access the actual test, there must be evidence that they 
spent time in the Practice Area. If there is not a record of 
their unique log-in ID, access to the test is denied. This is 
necessary to ensure that students are familiar with the Web 
site navigation and response system before they attempt to 
use it for the actual assessment. In addition to the Practice 
Area, there is a CATS Online DEMO area that is open for 
year-round preview and use by students, teachers, parents, and 
(see sidebar). 



The CATS Online Demo Area can 
be accessed by going to 

http://catsonline.ecollege.com/ 

Type in one of the following log-in 
IDs: 

Grade 4 - Login ID: g04st2161 
Grade 5 - Login ID: g05st9398 
Grade 7 - Login ID: g07st9567 
Grade 8 - Login ID: g08st4057 
Grade 10 - Login ID: g10st7314 
Grade 11 - Login ID: g11st6974 
Grade 12 - Login ID: g12st1215 

the general public 



The Universal Design of CATS Online 

The universal design of CATS Online is intended to give students a more 
personalized and equitable way to demonstrate what they have learned and to 
maintain the integrity of the assessment. This is accomplished by incorporating the 
three principals of UDE described earlier; (1) multiple means of presentation, (2) 
multiple means of expression, and (3) multiple means of engagement. 

M ultiple means of presentation. The contrast between CATS Online and 
previous versions of assessment is readily apparent — that is, between the static nature 
of paper- and-pencil test administration and the flexibility provided by use of the 
electronic format. A student using a computer screen is presented with just one 
question per screen, while a student using a paper version of an assessment has the 
challenge of focusing on one of five or more questions per page. The student re- 
sponse sheet may cause more confusion than the question booklet. Response sheets, 
with their densely compacted presentation of familiar bubbles for responses, can be 
overwhelming visually. Students using CATS Online click a radio button to respond 
to a multiple-choice item, while those answering on paper have to be sure they are 
answering the right question and matching it to the corresponding response bubble. 

When a student opens a paper test booklet, there are no immediate options for 
altering the presentation, whereas with an electronic interface such as CATS Online, 
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the first option given the student is to “Set Preferences” (see figure). This includes 
designating a preferred font size and type, which allows students to emphasize or 
enlarge type. Foreground and background colors can be individually determined, 
which is especially important for students with certain disabilities who benefit from 
alternative color schemes. 

Though students can set font preferences, some students have visual or literacy 
issues that prevent them from operating within a text-based environment. A number 
of studies have found computerized read-aloud to be an effective accommodation in 
this situation." The array of configurations of the read-aloud support in CATS 
Online further adds to the variety of presentation. Students can choose either a male 
or female voice, a “digital” voice, or the more humanlike “RealSpeak.” Of special 
interest is the option to set the speed of the voice. It has been reported that some 
students with Attention Deficit Hyperactivity Disorder (ADHD) prefer to set the 
voice speed at or near the highest setting of 250 words per minute (wpm), which is 
well over the typical human average of 150 wpm. An interesting question is whether 
flexibility of modulation might have special auditory benefits for some children with 
ADHD. This is certainly an alternative in presentation that would not likely be 
provided by a human reader and would only be available through an electronic 
medium. 
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While students who are auditory learners might benefit from this option, 
others who are visual learners may prefer to change how the read- aloud text appears 
on their screen. Students can choose to have the text highlighted and read directly 
from the screen or have the text placed in a separate text box. The synchronized text 
reading with dual color highlighting adds another feature to the visual dimension. 
Choices are also offered for how reading passages and questions appear. Some 
students choose to set the preferences so that a text passage and the questions appear 
on separate screens. Others prefer to have a split-screen setting with both the passage 
and the question presented in a side-by-side or top-to-bottom screen mode. 



The use of text-to-speech to assist students with decoding words is perhaps one 



of the most significant changes in presentation format and has the added benefit of 



being electronically constant. Concerns often arise about 
the possible differences in the way each human reader 
provides a read-aloud accommodation (e.g., intonation, 
accent, voice quality, etc.). The limitation of human read- 



FACT — 91 percent of students 
surveyed in the pilot study thought 
they scored better by taking the test 
online. 



Summary of CATS Online Multiple Means 
of Presentation 

1 . Students can select the font type and size and 
screen color scheme (contrast between background 
and foreground). 

2. Students can select the voice type, volume, speed 
and pitch, and pace of reading. 

3. Students can select how words are to be read (e.g., 
one sentence at a time or full screen) and how 
they appear (e.g., color of words, separate speech 
balloon or as appears in original textbox). 

4. Students can choose continued scrolling or dual 
frame (e.g., question on top and text response box 
on bottom or question and response box side by 
side). 

5. Design and delivery are user friendly (text reading 
vs. decoding, one question per screen instead of a 
page of questions, radio button response for each 
question vs. shading bubbles on an answer sheet). 



WHAT STUDENTS SAID— 

7 really like how the letters were 
colored and were big enough to 
read what they said. ” 

“I did not have to flip from page to 
page. That is how I would get 
messed up. ” 

FACT — 75 percent of teachers 
surveyed in the pilot study thought 
CATS Online was preferable to 
reading to the student. 

WHAT TEACHERS SAID— 

“This really helped my students feel 
confident and Independent. ” 

“From my observation of special 
needs students, a large majority of 
them thrive in a technology-related 
setting. Technology seems to be 
another learning style that many of 
them understand, where they might 
not typically understand the read/ 
written process as well. It brings to 
mind positive differentiation and 
alternative learning style tech- 
niques. ” 
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aloud accommodation has previously been cited as threatening the adequacy of this 
support and as possibly compromising the test validity.'^ 

M ultiple means of expression. CATS Online allows students to select a variety 
of strategies for interacting with the assessment. With a traditional, human read- 
aloud accommodation, it is often difficult for students to ask for questions, response 
options, or passages to be reread, especially when this support is rendered in a group 
fashion. Plus, some students are reluctant to ask for this type of additional assis- 
tance.'^ 



With electronic administration, students have more control and independence 
in determining how and when to use their mouse and/or keyboard, to read and 
reread questions or passages, and to respond. When completing open-response 
items, if consistent with students’ lEP and classroom accommodations, they may use 
word prediction to help them decide what words to choose to support their sentence 



development or conceptual intent. Students report special 
value in using the text reader to read back to them what 
they have written, as this allows them to reflect on the 
structure and meaning of their compositions and to use 
their keyboards to make changes as needed. Getting this 
kind of support with a human reader becomes difficult, 
and trying to correct or change written responses certainly 
is more cumbersome and messier. 

M ultiple means of engagement. Having the test 
read to students in a resource room setting or library 
creates many opportunities for distraction and loss of 



FACT — 84 percent of students 
surveyed in the pilot study used 
the computer to reread ques- 
tions. 

WHAT STUDENTS SAID— 

“I liked taking the test on a 
computer because everything 
i/vas at my own speed and I did 
not have to rush to get it done. ” 

“It was easy and I liked it be- 
cause I could do it myself. ” 



Summary of CATS Online Multiple Means 
of Expression 

1 . Students can use the keyboard and mouse to read 
and respond independently and selectively at their 
own pace. 

2. Students can use word prediction to facilitate 
completion of open response. 

3. Students can listen to what they have written, 
critically evaluate possible changes, and make 
corrections efficiently with the keyboard and 
mouse. 



FACT — 73 percent of teachers 
surveyed in the pilot study said 
their students reread questions 
more often online than when 
using a human reader. 

WHAT TEACHERS SAID— 

7 saw my students go back and 
read the passage several times 
for understanding. Many stu- 
dents will not do this when being 
read to by someone. ” 

“They enjoyed [and] appreciated 
the status of taking the test 
oniine. ” 



I m w a I Q H X • Volume 5 • © Edvantia 2005 • www.iaete.org/insight 



11 



focus, especially if the read- aloud is in a group format. 
Sitting in front of the computer screen with headphones, 
attending to one question at a time, helps students 



FACTS — 57 percent of students 
surveyed in the pilot study said taking 
the test online helped them pay 
attention. 



concentrate, both aurally and visually, on the task at hand 
free from the many sights and sounds around them. 

A comment heard over and over from students 
using CATS Online was “Now I can just do it myself!” 
Another frequent student comment was “I like being on 
the computer and not having someone read to me like a 
kid ”14 sense of independence, freedom, and control 
students taking the test on a computer have is vastly 
different from the feelings they seem to have when they 



63 percent liked being able to work at 
their own speed. 

83 percent said it was easier to 
concentrate when taking the test 
online. 

87 percent said they want to take the 
test online in 2006. 

WHAT STUDENTS SAID— 

“If I hadn’t taken the test on the 
computer, I would not have done my 
best at all. ” 



have to depend on a human read- aloud accommodation. 
While having to be read to by an adult in a separate 
setting seems to carry real stigma for some students, being 
able to say to their peers, “I’m taking my test on the 
computer,” affords a sense of status. 



Summary of CATS Online Multiple 
Means of Engagement 

1 . The computer screen offers additional visual 
cues, and headphones provide auditory 
engagement and reduce auditory distraction. 

2. Increased student control of assessment via 
technology increases student efficacy compared 
to traditional accommodations. 

3. Student self-concept and positive peer 
perceptions are enhanced by use of a computer 
accommodation. 

4. Taking tests on the computer is often perceived 
as “easier” and “more fun” by students than 
paper- and-pencil versions. 

5. Self-pacing and control of question/response 
review increases attention to content/ response 
and time devoted to test completion. 



“It helped you figure out stuff that I 
thought I didn’t know. It made me 
really use my brain. ’’ 

“It helps me pay attention to the test. ’’ 

“I got a headphone where it read to 
me online and that made me under- 
stand it a iot more. ’’ 

“When I took the test on the computer, 
I found it was easier to concentrate 
because I only heard what was being 
read on the computer. ’’ 

FACTS — 65 percent of teachers 
surveyed in the pilot study thought 
their students took more time to 
complete the test online than when 
using a human reader. 

72 percent of teachers said their 
students were more engaged with 
online assessment than with pencil- 
and-paper testing. 

WHAT TEACHERS SAID— 

“Computers are motivating for the 
students. ’’ 

“This is my second year using CATS 
Online, and i will never go back to 
paper-and-pencil testing as long as I 
have the available technology for my 
students. ’’ 

“Students seemed to try much harder 
and seemed to put much more effort 
into the test than with pencii and 
paper. ’’ 
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Technology creates a sense of engagement that is incomparable to what most 
students experience with paper-and-pencil test administration. The self-directedness 
of the accessible online assessment gives many students a feeling of empowerment 
that can facilitate a more accurate display of student ability. As put forth by 
Lichtenstein, “the search for independence and the struggle for autonomy”'^ are 
foremost in the minds of many adolescents. 

Impact of Universal Design on Student Performance 

Both tangible and intangible factors that accompany assessment in a univer- 
sally designed format appear to contribute to improved student performance. Recent 
research by Dolan, Hall, Banerjee, Chun, and Strangman found that computerized 
testing with text-to-speech support resulted in improved performance, especially on 
items with reading passages that exceeded 100 words. Kentucky school districts 
also reported improved student performance with the use of CATS Online.'^ 

Of critical importance is that 91 percent of students surveyed*® thought that 
they performed better on the state assessment by taking CATS Online (e.g., “1 just 
want to say that this was a very good thing to do because 1 think that 1 have done 
better this year on the test than any other year.”). The most frequent comments 
students made about taking the test online in Kentucky were that CATS Online was 
“easy” (e.g., “1 like to take the test on the computer. It seems so much easier and 1 
don’t have to have a person read to me.”) and “fun” (e.g., “1 get to type instead of 
write. What could be more fun?” “Taking the test on the computer is very fun and 
relaxing to me.”). In our experience, these descriptors have rarely, if ever, been used 
by students to convey their feelings about taking their state assessments in a tradi- 
tional paper-and-pencil format. While higher test scores are the most obvious 
indicator of improved student performance, it is worthy to note that heightened 
student engagement, greater independence, and more learner control via the online 
assessment appear to significantly contribute to helping students effectively demon- 
strate what they have learned. 

Next Steps 

While more longitudinal data collection and empirical analyses are needed to 
verify the impact of Kentucky’s online assessments on a large scale, the annual rate of 
increase in CATS Online participation of twofold or better over the past three years 
indicates strong movement from paper to computer administration in Kentucky. In 
fact, due to Kentucky’s success with online assessment for students with disabilities, 
the KDE expanded online assessment in 2005 to two other areas as described below. 

In February, the Kentucky Occupational Skills and Standards Assessment 
(KOSSA) was offered online for the first time to about 1,700 students in career and 
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Figure. Student view of practice area in continuous scrolling 
presentation mode 




technical education programs, and a pilot of the KCCT for the general population 
was conducted with almost 900 students. Plans are under way for further expansion 
of these assessments for 2006. It is worthy of note that the KOSSA online allowed 
use of text-to-speech (using the Read & Write software) for any student who used 
this tool as a literacy support in instruction. Although it was used minimally during 
administration of the KOSSA in 2005, the authors hope to expand the use of this 
tool in 2006 and beyond. They expect more and more students and school staff to 
realize how text-to-speech technology can support access to career and technical 
education content for a range of students, not just those with disabilities. 

Although some students have been afforded the opportunity to take the 
standardized assessment in the online format, not all students have been able to 
benefit from the accommodation. While Read & Write software is now available in 
almost every Kentucky school, recent investigation has shown that only about half of 
the schools are regularly using the technology as a literacy support due, in part, to 
technical support and training issues.'^ School districts also reported insufficient 
hardware with the capacity to run the software. 

Although incorporating technology does not come without issues, Kentucky is 
staunchly supporting the use of online assessments because, as one teacher said, 
“Testing online is the future. I am proud that its beginning included our exceptional 
children. It has proven that they can perform with their age-appropriate peers. 
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Online Assessments of 
Technology Literacy: 

The Perfect Petri Dish 

by Mary Axelson 



he path to assessing K-12 technology literacy has led to many surprises — 
and we’ve only just begun. Efforts to understand what students know about infor- 
mation and communications technology (ICT), for example, have prompted the 
development of assessments that involve computer-based simulations. Already, 
simulations of popular productivity software applications provide an environment 
for evaluating users’ skills in using the software itself But assessment designers are 
now brimming with not-so-apparent thoughts about how to harness such simula- 
tions in assessing content areas such as science, literature, and music. Improved 
communications between assessment and technology experts is fostering creative 
synergy as teams from both camps work together to develop an assessment of 
technology literacy. Meanwhile, the ongoing exercise of defining “technology 
literacy” may spin every state into a rewrite of content standards and place what 
many are calling “21" century skills” firmly in the curriculum. 

Technology literacy assessments may become the foundation for the next 
generation of all other online testing. Most states are hoping to move their annual 
high-stakes assessments online, and some have already done so. Some even hope to 
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create a much more comprehensive system of online formative, benchmark, and 
summative assessments. Online assessments enable a faster return of results and 
greater interoperability of the data with high-tech tools, but widespread acceptance 
of such assessments hinges on educators’ confidence that using a computer is not a 
barrier for a student’s ability to demonstrate what he or she knows. 

A rush of activity directed at assessing technology literacy is evident worldwide. 
The ability to use high-tech tools is commonly regarded as an individual’s founda- 
tion for lifelong learning and as a path to a nation’s economic health. In this regard, 
many fear that the United States is ailing. That prevailing impression, however, has 
no measure. At this time, the government has no way to determine how much of the 
labor force can use commonly available technologies, schools cannot determine if 
their technology programs are effective, and policymakers do not know if technology 
funds are well spent. 

The No Child Left Behind Act of 2001 (NCLB) gave impetus to the move- 
ment to evaluate technology literacy by stating a federal goal that all students will 
demonstrate technology literacy by the end of the eighth grade, beginning in 2006. 
Specifically, Title II, Part D Goal 2 (a) reads as follows: 

To assist every student in crossing the digital divide by ensuring that 
every student is technologically literate by the time the student finishes 
the eighth grade, regardless of the student’s race, ethnicity, gender, family 
income, geographic location, or disability. 

States have been given responsibility for defining what is meant by “technology 
literacy” and for developing a process to evaluate student proficiency in this area. 

The Ever-Expanding Definition of Technology 
Literacy 

Assessing technology literacy requires a precise definition of technology. The 
boundaries of technology could be limited to common software and hardware, set to 
include all information and communications technology (ICT), or expanded to 
include technologies beyond the realm of computing. 

In 2002, the National Research Council Committee on Technological Literacy 
authored Techni gI ly Speaki ng: W hy Al I Ameri cans N eed to K now M ore About T echnol- 
Ogy. This report defines technology not as simply computers, handhelds (PDAs), and 
other high-tech tools but as all tools and processes with which humankind adjusts 
the natural environment — from a shovel to the process by which scientists identify 
chromosomes on a strand of DNA. 
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Greg Pearson led the committee in creating both iGChnically 
Speaki ng and an upcoming report on assessing technology literacy. In 
an e-mail-based expert panel sponsored by the Institute for the 
Advancement of Emerging Technologies in Education (lAETE), he 
observed that equating technology literacy with ICT literacy is not 
wrong, just narrow. He writes: 

The Academies are far from the first to describe and 
promote this larger view of technology. Philosophers of 
technology, such as Langdon Winner, historians of tech- 
nology, like Thomas Hughes, and technology critics, like 
Neil Postman, have all taken this larger view. So has the 
STS (science, technology, and society) movement in U.S. 
post-secondary education, and the International Technol- 
ogy Education Association, which represents technology 
education teachers in this country. 

The word I i teracy has a correspondingly wide range of possible 
meanings, from mastery of basic skills, to fluency, to advanced levels 
of proficiency and creativity. Literacy, in iGChniGlIy Spoaki HQ, includes 
the ability to use not only relevant technology tools but also critical 
thinking skills, including being able to analyze risks and benefits on 
issues ranging from genetically modified foods to oil drilling in 
environmentally sensitive areas. This definition of technology literacy 
seeks to guarantee informed participants in the democracy and the 
economy. For the scientific community, literacy is tied to the essential 
ability to apply the process of scientific inquiry. For others, it is closely 
tied to rhetoric as an academic discipline' and therefore implies a 
certain facility in responding to text and other forms of media. This 
information age literacy requires a level of discernment that was once 
considered an elitist part of the K-12 curriculum. 

However, the Internet’s simultaneous glut and wealth of re- 
sources requires sophisticated discernment in determining the reliabil- 
ity and accuracy of information. Workforce uses of ICT also require 
skills in communicating, analyzing and interpreting data, problem 
solving, and collaboration. While these skills may not be new, the 
tools that support them continually change. These changes — and the 
sense of immediacy the tools bring to information gathering and 
sharing — do require new skills. Indeed, millennial literacy may be 



The concept of technol- 
ogy competency or 
proficiency became 
popular when microcom- 
puters were introduced in 
the nation’s ciassrooms in 
the 1980s. This concept 
concerns basic skiiis and 
knowiedge reiated to 
hardware and software. 
Eariy courses on computer 
skiiis often focused on 
basic operations, troubie- 
shooting, and program- 
ming in ianguages such as 
BASIC. 

21st Century Learning 
Skills, as described by the 
Partnership for 21st 
Century Skiiis, can be 
divided into three catego- 
ries: thinking and probiem- 
soiving skiiis, information 
and communication, and 
interpersonai and seif- 
direction. 

ICT literacy links 21st- 
century skills to information 
communication technology 
(ICT) tools— tools that can 
change over time— to 
support the development 
of and demonstration of 
critical thinking skills and 
higher-order cognition. 
Examples given of ICT 
literacy, according to the 
Partnership for 21st 
Century Skills, include the 
use of ICT to solve 
problems, to demonstrate 
creative ideas or thoughts, 
to access and evaluate 
information, or to enhance 
productivity and personal 
development. 
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interpreted as the ability to teach oneself within an environment of ever-changing 
tools and protocols. Precisely who is to be held accountable by the assessment of 
these expanded sets of abilities remains undefined. 

What is taught in schools today is driven by content standards. Current 
definitions of ICT literacy include a complex web of skills and knowledge that 
incorporate information addressed by content standards as well as higher order 
thinking and performance skills that require the use of ICT tools used in many 
current and future careers. The assessment of ICT literacy must find a niche within 
the core content areas in order to be addressed in schools. 

Do ICT skills, though, require a separate test? One approach to addressing this 
concern would be to have a baseline technology check included in a subject- area 
assessment to ensure that difficulties in using the technology do not hamper a 
student’s ability to demonstrate content knowledge. Ensuring that students have 
basic technology skills is a prerequisite to teaching and assessing complex learning 
skills. Schools will need to address this concern if they are to fully utilize digital tools 
that allow for the collection, analysis, evaluation, synthesis, and creation of informa- 
tion across multiple media. 

State directors of education technology have stepped up to assume responsibil- 
ity for responding to the NCLB goal so far as it relates to technology proficiency. 
Even with that common goal, definitions are difficult. The State Education Technol- 
ogy Directors Association (SETDA) gathered a technology literacy assessment (TEA) 
work group to define technology literacy. Jerry Bates, a participant from the Tennes- 
see Department of Education, wrote: 

With technology in the vanguard of virtually every social enterprise 
today, I knew we would bring a range of vested interests to the table. But 
what amazed me the most was how difficult it became to define the 
elephant in the center ring! 

Those who have examined the issue seem to agree that addressing technology 
literacy in the schools will require more than ensuring equitable access to technology. 
It will require more than increasing the availability of hardware and access to, as well 
as safe and ethical use of key applications. Knowing how to surf the Web doesn’t 
equate to expediently finding and capturing the reliable and credible information 
one is looking for. Will a student be fooled by slick graphics and authentic-sounding 
information created and posted by an individual posing as a nonprofit research 
center? Being able to copy and paste doesn’t equate to being able to organize infor- 
mation in different formats and to communicate effectively. Will students know the 
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rights of use for information they find and how to safeguard their personal informa- 
tion? Knowing how to reboot a locked computer or connect to a fileserver doesn’t 
equate to evaluating and procuring hardware and software to meet the ever-changing 
needs of a lifelong learner. As Educational Testing Service (ETS) observes, 

We will surely one day succeed in closing the “digital divide” that 
separates the technology haves from the have-nots. But we will not 
succeed by merely providing access to equipment: we must also provide 
the intellectual tools to use technology effectively, in school and in the 
workplace.^ 

Similarly, the International Technology Education Association, which pub- 
lishes Standards for Technologiral Literacy: Content for the Study ofTechnology, explains 

that 

technological literacy is far more than the ability to use technological 
tools. Technologically literate citizens employ systems-oriented thinking 
as they interact with the technological world, cognizant of how such 
interaction affects individuals, our society, and the environment. Techno- 
logical literacy is the ability to use, manage, assess, and understand 
technology.^ 

SETDAs TLA work group arrived at a definition of their “elephant” as follows: 

The ability to responsibly use appropriate technology to communicate, 
solve problems, and access, manage, integrate, evaluate, and create 
information to improve learning in all subject areas and to acquire 
lifelong knowledge and skills in the 21st Century.^ 

This definition, combined with the National Educational Technology Stan- 
dards for Students (NETS-S) from the International Society ofTechnology Educa- 
tion (ISTE), is the foundation that most states are using for meeting the eighth- 
grade goal mentioned in NCLB. The North Central Regional Educational Labora- 
tory (NCREL) is in the process of building rubrics for NETS-S that educators can 
use to assess technology literacy.^ Some NCREL NETS-S rubrics are completed and 
others are open to comments. 

SETDA charts the status of each state in four categories: definition of eighth- 
grade technology literacy, policy for standards, process of evaluating progress, and 
reporting requirements and methodology. All are works in progress. Some states 
embed technology literacy standards in the core content standards; others create 
separate standards for technology literacy, and some states have no technology 
literacy standards at all. 
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Standards That Keep Pace with the Rate of Change 

Technology changes quickly. Standards that specify a software program, or 
even a particular computing device, are likely to become obsolete in a hurry. For this 
reason, many technology standards (both state and national) have been written to 
focus on tasks rather than tools. Thus, ISTE’s NETS-S reads, “Students use technol- 
ogy to locate, evaluate, and collect information from a variety of sources.” Quite 
purposely, there is no mention of the Internet or other specific digital archives. 

Similarly, a fifth- through eighth-grade NETS-S science standard reads “Use appro- 
priate tools and techniques to gather, analyze, and interpret data.” For standards to 
withstand the shifting sands of technology, they must include the ability to recognize 
the most suitable tool for the job (see sidebar for the complete “Technology Founda- 
tion Standards” from ISTE NETS). 

Technology Foundation Standards for Students 

1 Basic operations and concepts 

• Students demonstrate a sound understanding of the nature and operation of 
technology systems. 

• Students are proficient in the use of technology. 

2 Social, ethical, and human issues 

• Students understand the ethical, cultural, and societal issues related to technology. 

• Students practice responsible use of technology systems, information, and software. 

• Students develop positive attitudes toward technology uses that support lifelong 
learning, collaboration, personal pursuits, and productivity. 

3 Technology productivity tools 

• Students use technology tools to enhance learning, increase productivity, and 
promote creativity. 

• Students use productivity tools to collaborate in constructing technology-enhanced 
models, prepare publications, and produce other creative works. 

4 Technology communications tools 

• Students use telecommunications to collaborate, publish, and interact with peers, 
experts, and other audiences. 

• Students use a variety of media and formats to communicate information and ideas 
effectively to multiple audiences. 

5 Technology research tools 

• Students use technology to locate, evaluate, and collect Information from a variety of 
sources. 

• Students use technology tools to process data and report results. 

• Students evaluate and select new information resources and technological 
Innovations based on the appropriateness for specific tasks. 

6 Technology problem-solving and decision-making tools 

• Students use technology resources for solving problems and making informed 
decisions. 
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Emerging Assessments 

The phrase “21st century skills” has emerged as an accepted descriptor for the 
advanced skills and knowledge that use, and often rely upon, ICT tools. These skills 
were the focus of an international ICT literacy panel convened by ETS. The panel’s 
2002 report identifies the desire of Statistics Canada, the National Center for 
Education Statistics (NCES) in the United States, and the Organisation for Eco- 
nomic Co-operation and Development (OECD) in Paris to include sophisticated 
ICT literacy skills in their international assessments of students and adults. Says the 
Educational Testing Service, “To date, however, no one has put forth a framework to 
assess if an individual has achieved ICT competency to function successfully in a 
knowledge-based society” (www.ets.org/Media/Research/pdf/ICTREPORT.pdf). 

As reflected in a 2005 report from the Partnership for 2T' Century Skills 
(P21), the intervening three years have seen the introduction of some new models. 
The P21 report. Assessment of 2D Century Skills The Current Landscape, identifies 
numerous assessments that target complex cognition within the context of using 
ICT tools. P2 1 is an organization formed in 2002 to define the new abilities re- 
quired for our high-tech information age. It came together with support from the 
U.S. Department of Education, the National Education Association, and numerous 
corporations and their educational programs (AOL Time Warner Foundation; Apple 
Computer, Inc.; Cable in the Classroom; Cisco Systems, Inc.; Dell Computer 
Corporation; Ford Motor Company; Microsoft Corporation; and SAP among 
others). These entities have worked to define 21st century skills; to encourage the 
incorporation of 21st century skills within state content standards; and to advance 
the nation’s ability to assess the complex cognition required for problem solving, 
collaboration, and understanding economic systems. 

P21 has identified the key goals for providing a 21st century education as 

follows:® 

• Emphasize core subjects. 

• Emphasize learning skills (information and communication skills, thinking 
and problem-solving skills, and interpersonal and self-directional skills). 

• Use 21st century tools (ICT tools) to develop learning skills. 

• Teach and learn in a 21st century context (global awareness, financial, 
economic and business literacy, and civic literacy). 

• Teach and learn 21st century content. 

• Use 21st century assessments that measure 21st century skills. 
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Clearly, interrelationships weave an intricate network, but separate assessments target 
each. The June 2005 draft of P2Ts assessment report, speaking to assessment of the 
broader notion of ICT literacy, explains that 

. . . interest has grown in developing assessments that (1) reveal the 
cognitive skills that students employ in conjunction with their use of 
technology and (2) use technology as their means of delivery. The process 
of creating such assessments is only beginning even among the leading- 
edge education ministries, with the exception of the United Kingdom. 

Developing Models of ICT Assessment 

Described below is new assessment work from Britain’s Qualifications and Curricu- 
lum Authority (QCA), the North Carolina Department of Public Instruction, the 
Florida Department of Education, ETS, Learning.com, and the International 
Computer Driving License (ICDL). The focus of the assessments range from 
technology literacy to the more expanded notion of ICT literacy, but all use simu- 
lated software environments for an on-screen assessment. Lessons learned from each 
illustrate possibilities for the development of new types of assessments, as well as 
assessments that measure more complex skills. Additionally, the National Center for 
Education Statistics’ NAEP (National Assessment of Educational Progress, also 
referred to as the Nation’s Report Card) intends to explore the creation of such an 
assessment for 2012. 

The UK’s Key Stage 3 ICT Literacy Assessment 

The United Kingdom is currently piloting its Key Stage 3 (ages 11-14) ICT 
Literacy Assessment.^ The effort stands out as a model for computer-based testing of 
technology literacy, with the goal of assessing ICT literacy in the future. Created by 
the British government, it uses a simulation-based environment — a virtual town 
dubbed Pepford — to assess students’ technology skills and their ability to apply 
computer-based tools to common problems. An adaptive test (combined with a 
teacher’s estimation of a student’s skill level) determines students’ entry level and 
assigns a series of tasks. For example, students may be placed in the role of movie 
theater employee and asked to prioritize needs, assemble documents, and schedule 
events by communicating via simulated e-mail and to report information they have 
found and synthesized in a Gantt chart. One task, for example, might require the 
student to find the theater’s seating chart online. These tasks are tackled with what 
the creators of the simulated computer environment call a “walled garden” of 
applications, Web sites, search engines, and other resources that allow students to 
demonstrate their abilities without accessing the Internet. 
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The environment allows students’ actions to be closely tracked and analyzed at 
a depth unparalleled by any other assessment used in K-12 schools today. The 
resulting reports are used as a way to identify students’ individual strengths and 
weaknesses and as wider summative measures of school performance. Explains 
Martin Ripley, director of e-strategies for QCA: 

The use of simulations is a key development. A simulator provides the 
context within which authentic assessment tasks can be designed 
and delivered to students. It also facilitates the development of assess- 
ment tasks which invite students to combine a range of capabilities and 
skills. The combination of these two aspects enables us to assess higher - 
order information technology capabilities — such as choice or communi- 
cation. 

For scoring purposes, the software records and scores the actions that the 
student takes while completing the test. For example, a higher order 
capability for a 14 year old student in England includes designing a 
system for someone else to use. That capability in turn includes an 
assessment of the end-users requirements. This we assess dynamically in 
the virtual world of Pepford by collecting evidence of the student re- 
searching into those requirements, by sending and receiving e-mails to 
ask about requirements, by the student refining their system to meet 
requirements and so on. To achieve this we have worked with an exten- 
sive range of teachers to document the processes that students go through 
when producing eloquent or satisfactory responses to the task set. We use 
this to create a matrix of plausible routes that a student will take when en 
route to complete a satisfactory (or better) response.^ 

The on-screen test is not technically online in terms of being delivered over the 
Internet from a central server, and its creators don’t know that it ever has to be. The 
pilots and first rounds of delivery use software distributed to the schools located on 
and delivered from a school-based server. Ripley explains that schools’ Internet 
connections are not reliable enough to guarantee that students will not lose access — 
or, worse, lose data on their work. 

Ripley wants to link the Key Stage 3 test environment with “the broader 
notions” of 2T' century literacy. He clarifies, “And I don’t mean technological 
literacy. I mean citizenship, functional literacy, numeracy, problem solving, indepen- 
dence, self-learning, motivation, all of those sorts of abilities.” All are areas of 
knowledge, says Ripley, that “paper-based assessments just can not reach.” He also 
wants to expand the test environment to include items such as PDAs, cellular 
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phones, probeware, and other devices that students might have access to at home or 
in subsequent careers. “One of the key challenges for us in this test development 
project,” explains Ripley, “is to extend the simulator well beyond its current focus on 
a PC desktop environment, to include other technologies and to assess a wider range 
of technological capabilities” (for more information, visit www.rm.com/qca/ 
mainpage.asp?cententref=QCAP368909). 



Figure 1 . 

Example Problem from Key Stage 3 ICT Literacy Assessment 




The North Carolina Online Test of Computer Skills 

North Carolina’s Department of Public Instruction (NCDPI) has been assess- 
ing technology literacy since 1996, and passing the exam has been required for high 
school graduation since 2001. In October of 2005, the department implemented its 
online assessment of the new 2004 technology standards, the North Carolina Online 
Test of Computer Skills. This involved testing every one of the state’s approximately 
100,000 eighth graders. The online assessment greatly simplified the process by 
replacing two separate tests that students had been required to pass, one multiple 
choice and one performance based. 

For the performance-based assessment, explains Mildred Bazemore, section 
chief for the Test Development Section of the NCDPI, students had to be at a 
computer with their paper- and-pencil answer sheet. Administering this assessment 
was a challenge. Various schools within a district were using computers and operat- 
ing systems that ranged from an Apple He to a desktop equipped with the most 
recent version of Microsoft Windows. Schools also used various brands of productiv- 
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ity software (word processing programs, spreadsheet software, and databases). As a 
result, the state had to make 67 different applications available for every testing 
event. The costs and frustration of those logistics, and the endless paper stream of 
hand-scored assessments, made the online option stunningly attractive. Indeed, any 
online version of an assessment holds promise for an improvement to logistics. In a 
personal communication with the author, Bazemore explains: 

Were constantly exploring ways of doing things more effectively and 
efficiently. I don’t know if you’ve ever seen a testing warehouse with all 
the forklifts, and the test books, the pieces of paper, the dust mites, the 
paper mites and so forth. In North Carolina we have 1 1 5 districts and 
probably just under a hundred charter schools, and all of these pieces of 
paper have to go in all these different directions. It’s very costly, very time 



consuming. There’s too much touching of the 

that you can’t control. 

NCDPI’s commitment to assessing technol- 
ogy literacy stems from the conviction that such 
literacy is an essential foundation for lifelong 
learning (see sidebar). Its project benefited from the 
state having had previous experience with design- 
ing an online assessment. North Carolina has fully 
developed an adaptive assessment that was in- 
tended to be used for state accountability purposes 
by students with learning disabilities. Adaptive tests 
are not currently used for federal reporting require- 
ments because they limit the number of standards 
presented to a student and not all students may see 
questions related to the same set of standards; thus 
that particular assessment is not in use at this time. 
Bazemore adds that the state has had the technical 
know-how to deliver assessment online since 200 1 , 
but infrastructure at the school level has only 
recently been able to receive it. Technically, some 
schools are still not able to do so. For these schools 
and for students needing assistive technology, 
NCDPI has created two portable labs. NCDPI has 
also created an alternate technology assessment that 
is not computer based. 
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North Carolina’s Definition of Eighth- 
Grade Technology Literacy 

In keeping with the philosophies and 
content of the North Carolina Standard 
Course of Study and the ABCs of Public 
Education, technology will nurture and 
empower the development of students to 
become self-directed, life-long learners; 
complex thinkers; quality producers; 
collaborative workers; community 
contributors. Students will learn how to 
select, evaluate, and use a variety of 
technology applications and resources 
for their personal and academic needs. 
Through the acquisition of skills and 
knowledge, students will have the ability 
to participate and thrive in the American 
economic and political systems. Adminis- 
trators and teachers will use technology 
to address the learning styles of today’s 
students more effectively. 



Excerpted from a SETDA chart of state 
responses to assessing technology 
literacy, www.setda.org/Toolkit2004/ 
evaluation 06 State of States.htm 



For the technology literacy assessment, the state worked closely with North 
Carolina State University to build a proprietary software environment known as 
NCDesk. This collection of programs, developed in the programming language Java, 
replicates common applications such as word processing, spreadsheets, databases, 
e-mail, and file management. Additionally, NCDesk includes a test interface and a 
test simulation that are used to teach students how the testing interface works. 

A test simulator movie is available online (cskills.ncsu.edu/ncdesk/ 
simulationdemomovie.asp). It features such items as a request to insert a picture of a 
car in the home page of a video company, a multiple-choice question to identify the 
best search terms, and a multiple-choice question asking for the most recent infor- 
mation on a spreadsheet of sales for five different video games. The actual online 
assessment requires about two hours, including administrative tasks and breaks. All 
72 items are new and reflect the 2004 computer skills curriculum (54 items consti- 
tute the assessment and 18 are being field-tested). 

NCRegistration is an administrative tool built to accompany NCDesk. The 
application manages user access, registers students in bulk or individually, handles 
scheduling of test sessions, provides Student Information Questions (SIQ) for 
additional data collection, and issues reports. The assessment resides on a central 
server, and student responses are reported directly to the server as students move 
between questions. The feature minimizes the risk of losing data should a power 
outage or system crash interrupt the assessment. 

The development process included a feasibility study and trial with adults at 
volunteer sites in the fall of 2003. An expanded study and trial in the fall of 2004 
included a minimum of 10 locally chosen students per school, for a total of 5,620 
students starting the test at 608 schools. Of these, 4,783 students, or about 85 
percent, finished. A spring 2005 field test sampled the population of schools and 
students. In a window that ran from April 1 1 to June 15, 8,5 1 0 students were 
tested. With 6,361 starts and 6,198 finishes, the completion rate increased to 
approximately 95 percent. Jim Kroening, program manager for Performance Assess- 
ments within NCDPFs Test Development Section, attributes that rise to an in- 
creased awareness of the technical requirements, schools responding to those require- 
ments, and improvements in the technology and the delivery of the assessment at the 
central level. All testing rounds were followed by further development and debug- 
ging of the technology, test environment, and items. A technical readiness survey 
confirmed that every school had computers and sufficient bandwidth to administer 
the assessment. Multiple forms of the test provide additional security should stu- 
dents’ eyes wander around the computer lab. 
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To address online opportunities for students with special needs, Bazetnore is 
working to add a read- aloud feature to the technology assessment. This feature has 
not been affordable within their Java environment as text-to-speech software is often 
highly specific to the applications with which it can operate, and Java is not as 
widely supported as the basic programming language of the Web, HTML. She also 
wants to move other alternate assessments online and expand the simulation items to 
other disciplines, such as science. 

Looking back, Bazemore advises other states to seek a top-down approach for 
such a project, stopping short of actually recommending the effort to be legislated. 
Theirs was a grassroots approach involving schools and the test development depart- 
ment at NCDPI, and Bazemore says they were fortunate to gain support from the 
legislature via individual line items. Being able to respond to a state initiative, says 
Bazemore, would eliminate considerable red tape. 



Figure 2. 

Example Problem from North Carolina Online Test of Computer Skills 




The Florida Inventory for Teacher Technology Skill 

In April of 2006, the Florida Department of Education will begin field-testing 
its online inventory of technology skills for eighth graders. Last summer the depart- 
ment rolled out the predecessor of this tool, the Inventory for Teacher Technology 
Skills. 
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Kate Kemker, director of Instructional Technology for the Florida Department 
of Education, explains that the state’s 67 districts were looking to the state for 
assistance in understanding the technology proficiency levels of their teachers and 
students from a skills-based level. Kemker’s team started with the teachers. “We said, 
let’s really develop a tool that’s going to be performance-based and simulated, and we 
can really get at what the teachers know,” recalls Kemker.^ This decision propelled 
the development team to delve into the more complex notion of ICT literacy, rather 
than limit its work to basic technology skills assessment. The finished product 
simulates common productivity and communication software applications and 
includes many situational and multiple-choice questions. 



Figure 3. 

Example activity from Florida’s Inventory for Teacher Technology Skills 




Florida began its development process by focusing on ISTE’s NETS for Teachers 
(cnets.iste.org/teachers/t_stands.html): 

• Technology Operations and Concepts 

• Planning and Designing Learning Environments and Experiences 

• Teaching, Learning, and the Curriculum 

• Assessment and Evaluation 

• Productivity and Professional Practice 

• Social, Ethical, Legal, and Human Issues 
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The test development process took a year and half. In the end, Florida had both an 
assessment and a robust assessment environment that the department could 
customize to assess various content areas or use for different versions of the same 
content, such as an assessment of students’ ICT literacy. The Florida Department of 
Education expects that this will allow them to roll out the eighth-grade assessment 
as early as April 2006. Kemker explains that some areas need refinement — tasks 
with spreadsheets, for example, are proving to be problematic — but she has great 
confidence in the programming of the software shell. 



Districts are not required to use the online assessment, but 3,000 teachers had 
already taken the assessment by the fall of 2005. Kemker and Florida education 



administrators were enjoying a “data rush.” Patterns were emerging that will help 
districts to identify clear professional development needs, and individual teachers 



were designing their professional development plans based on the measures of 



proficiency reported by the assessment. Though professional 
development is the district’s responsibility, Kemker is working to 
tie assessment results to recommended resources. 

The student version of this technology literacy assessment, 
targeted at the eighth-grade level, begins with a clear goal: Florida 
plans to eventually have its high-stakes assessment, the Florida 
Comprehensive Assessment Tests (FCAT), online. At that time, 
says Kemker, “We can not let technology be the barrier. So we 
need to make sure that students have those skills, that they’re 
comfortable with using the computer; that when they go to take 
that online assessment, that it is not any barrier to their success.” 

The current assessment for teachers, explains Kemker, 
measures only basic technology skills, not technology integration. 
Similarly, the eighth-grade assessment is confined to computer 
skills and does not address the more complex cognition that can 
accompany them, such as using computer skills to analyze, 
synthesize, evaluate, or create information. 



The Pre-Invented 
Wheel 

“We have an incredible 
tool,” says Kate 
Kemker, describing 
Florida’s Inventory for 
Teacher Technology 
Skills. What’s more, it 
was created with 
federal funds, so the 
state hopes to make it 
freely available to 
others. The structure of 
such licensing, and 
perhaps hosting, has 
not yet been deter- 
mined, but inquiries are 
welcome. 

North Carolina has had 
numerous inquiries 
from districts and 
states about licensing 
its technology literacy 
assessment, but the 
state has had no time 
to consider this stage 
of their project. 
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Figure 4. 

Example activity from Florida’s Inventory for Teacher Technology Skills 



Description For this task you will use an Internet browser to view a middle-school class’ History projects. 




Reports from this semester 
Sludent Reports from earlier semesters 




The ETS ICT Literacy Assessment 

The ICT Literacy Assessment from ETS also provides a simulated environ- 
ment for authentic, on-screen tasks. This assessment is designed for higher educa- 
tion students and can potentially double for assessing educators’ ICT literacy. Terry 
Egan, ICT project manager, says ETS has received many requests to extend the 
work to K-12. 



One example of a task for postsecondary students follows: 

You’ve volunteered to create a flier for a community clean-up day to 
be held in your neighborhood. Include the map below, along with 
the following information, and create an attractive one-page flier for 
the event. The event will take place on Saturday, May 6, from 1:00 
to 4:00 p.m. at Lincoln Square Park. Event organizers want a tear-off 
sheet to print names, addresses, and phone numbers. 



A simulated document layout program is provided to help students complete the 
task shown in the example (create a flier), and the final product is scored for accu- 
racy and completeness. Explains the ETS report, “Additional scoring points might 
include evaluating the layout and inclusion of graphic design elements such as 
borders and lines.” Human scorers make those determinations. 
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Another task gives the scenario of a sibling with a “rupture of the anterior 
cruciate ligament,” and the task is to “find several reliable sources on the Web that 
recommend treatment and rehabilitation for this condition.” Asked how devious the 
assessors get in terms of replicating the vast amount of incorrect or misleading 
information on the Web as a form of distracter, Egan laughed as she explained that 
test authors don’t try to trick anybody However, the range of Web sites that can be 
returned from a search can include off-base personal Web sites, advertisements, and 
dated medical articles, as well as current information that has been written or 
reviewed by experts. 

Before creating their ICT literacy assessment, ETS convened an international 
panel to “examine the need for a measure of ICT literacy across countries and within 
specific organizations, such as schools and businesses” and “to develop a workable 
fram0/VOrk for ICT litoracy” that could be used internationally. The resulting report, 

D igital T ransformation: A Framework for I CT Literacy, concludes that ICT literacy is 
“a continuum of skills and abilities” and that “the concept of ICT literacy should be 
broadened to include both critical cognitive skills as well as the application of 
technical skills and knowledge.” A desire to address “the digital divide” and to “help 
technology fulfill its potential transformative role” motivated the broad approach. 

Participants on the international panel arrived at a common definition of ICT 
literacy: “ICT literacy is using digital technology, communications tools, and/or 
networks to access, manage, integrate, evaluate, and create information in order to 
function in a knowledge society.” Explains Egan, “The construct, or the framework, 
or the definition of Critical thinking skills — there’s not a lot of agreement on it. But, 
without a doubt, when people see the types of tasks we have on the assessment, they 
recognize immediately that it involves problem solving and critical thinking.” The 
environment that simulates common digital technologies, such as communication 
tools and productivity software, helps students demonstrate these higher order skills 
within the context of a technology-supported environment. 

Learning.com’s TechLiteracy™ Assessment 

Learning.com, a company in Portland, Oregon, is in the process of developing 
assessments of technology literacy that are a direct response to the NCLB goal. The 
company has created two assessments — one for grade five and another for grade 
eight — and aligned them to ISTE NETS-S and its performance indicators. Kris 
Homsi, manager of product strategy, explains that the company focused on perfor- 
mance indicators rather than the actual NETS-S standards. The company also found 
the performance indicators of Texas, New Jersey, and North Carolina to be “well 
operationalized” guidelines and used them as guidelines in their development of 
their assessment. 
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The assessments at both grade levels include a mix of simulation-based activities 
and multiple-choice items, and they are designed to be administered in a single, 50- 
minute class period. The scope is limited to what Homsi identifies as “foundation 
skills” organized around seven modules: 

1 . word processing 

2. telecommunications/network (includes Web browsing and e-mail skills) 

3. multimedia/presentation (audiovisual tools; graphics tools; presentation 
software; and, for the elementary version, visual mapping software) 

4. spreadsheets 

5. databases 

6. social and ethical issues 

7. systems and fundamentals (including basic software operations, file 
management, navigation, and other system-level tasks) 

Learning.com has also developed a technology skills curriculum program known as 
EasyTech. Eventually, this and other resources will be tied to results for remediation. 

One possible advantage of Learning. corn’s solution is that the company hosts 
the administration and reporting capabilities. That limits the technical burden on 
schools and generally requires less technical support. When the latter is needed, the 
company can supply it. Schools do need to ensure that they have adequate band- 
width, operating systems, browsers, and sound and animation drivers. For more 
information on Learning. corn’s TechLiteracy Assessment, visitwww.learning.com/tla. 

Figure 5. 

Example problem from Learning.com’s TechLiteracy™ Assessment 
7 Usft IhB Au'4f(1 camfliand ^ nslu lha cchimti 
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The International Computer Driving License 

The International Computer Driving License (ICDL), a for-profit company, 
offers adults and students certification with a series of tests on basic computer skills 
rather than ICT literacy. There are seven module tests. The first (concepts of infor- 
mation technology) uses a multiple-choice format, but modules two through seven 
are performance-based items within simulations of applications. The other modules 
are titled Using the Computer and Managing Files, Word Processing, Spreadsheets, 
Database, Presentation, and Information and Communication. The tests are admin- 
istered online in proctored testing centers. 

The syllabus is created by the European Computer Driving License Founda- 
tion Ltd. (ECDL-F), a nonprofit organization that is the global governing agency for 
ECDL certification. Outside of Europe, the same program is identified as ICDL. In 
all, it is used in 138 countries, with a single licensee managing it within each 
country. 

In the summer of 2004, ISTE, Vantage Learning, and ICDL-US, which 
licenses the ICDL certification, announced their intention to build an assessment as 
a direct response to the NCLB goal for assessing technology literacy in the eighth 
grade. However, explains Heather Shay, vice president of education for ICDL-US, 
subsequent research did not show an adequate market for such an investment, and 
the partners shelved the project (for more information, visit www.icdlus.com). 

Making the Assessments a Reality 

The authors and developers of all of these assessments are confident that 
simulations can help them better understand each student’s understanding of a 
topic. The research base to support that impression, however, is thin, and this is a 
barrier to investment. States have encountered difficulties moving a multiple-choice 
test to an electronic environment, one being that differences in the look or amount 
of information displayed on the screen can alter performance. Oregon found that 
some items in their online assessment perform differently with certain student 
populations, but the state hasn’t yet “cracked the code” on that mystery." Addition- 
ally, most educators and decision makers grew up in an era of paper-based assess- 
ments with familiar multiple-choice response formats. The uncertainty of not only 
using different media but sometimes radically different questioning formats can be 
unnerving to some. 

The interplay between cognition and high tech tools is a complex, chicken- 
and-egg issue. As media theorist Douglas Rushkoff asks, “Does using Microsoft 
Word lead me to write differently than when I’m using Eudora? Why? And how does 
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a particular program lead to a particular set of behaviors or values? If I use Quicken, 
will I start to think that balancing my checkbook is more important? If I trade 
online, will I make more trades than if I have a broker I talk to on the phone?” 

Martin Ripley, referencing all forms of “e-testing,” including simulation, 
explains: 

I like to remind colleagues that paper-based testing has been in use for 
140 years. The science and business of paper-based testing is well refined 
and well evidenced. A person experienced in managing, researching or 
administering one set of paper-based tests is likely to be able to move in a 
straightforward way to a similar role in relation to other paper-based 
tests. In contrast, there is little stability yet in e-testing. The business 
processes are immature. Even when the e-test consists of closed response 
or multi-choice items, the on-screen delivery involves making design 
decisions on the basis of emergent thinking, not on industry best practice 
standards (which do not yet exist). 

Speaking specifically of simulation, he observes, “The nature and extent of innova- 
tion in this type of redesign challenges psychometric expertise and wisdom.” Greg 
Pearson, in the midst of preparing a National Academies report on assessing technol- 
ogy literacy, observed, “The assessment literature is mostly silent on simulation and 
essential psychometric issues such as reliability, validity, and precision.” 

Pearson identifies three circumstances essential to expanding the research base: 
(1) having a critical mass of opinion within the relevant communities (e.g., simula- 
tion developers, educators, assessment designers, and policymakers) that such 
research is needed, (2) having enough people with the requisite backgrounds and 
skills to do the research, and (3) funding. He points out that there is a tremendous 
amount of work exploring computer simulations in areas outside of K-12, such as 
that performed by the Department of Defense and the training industry, and asserts 
that these are resources of value. 

Simulation work is also happening within the K-12 sector, but it tends to focus 
more on learning than assessment. One notable project, Multi-User Virtual Environ- 
ment Experiential Simulator (MUVEES) (muve.gse.harvard.edu/muvees2003/ 
index.html), creates a simulated, collaborative environment that multiple students 
can use to figure out the source of illness in a late-nineteenth-century town. Chris 
Dede, the lead researcher on the project, is carefully exploring what software devel- 
opers call the clickstre0m — the vast range of information that can be gleaned by 
choices that students make. This simulation makes it possible to record and analyze 
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mouse movements and where a student is looking on the screen. The diagnostic 
possibilities are astounding. 

Measuring technology proficiency is a goal and a reporting requirement of 
NCLB. If states are motivated by this requirement to go beyond basic proficiency to 
address the complex skills and knowledge supported by and demonstrated by ICT 
tools, then states may pursue these opportunities and generate the needed research. 

It will require patience. Ripley explains, “We started our project to develop the ICT 
test for I3-year-oIds in 2001. We aim to have it rolled-out to all of our secondary 
schools (high schools) by 2008. I do not believe that we could achieve our aims in 
less than seven years!” 

Moving Beyond the Tag Team Development Process 

Ripley observes that QCA’s core team for designing a computer-based assess- 
ment “numbers around 15-20 and can be led effectively as a single team. There is no 
parallel in my experiences with paper-based tests, which are often developed in a 
linear process, with different teams (involving large numbers of people) responsible 
for the various stages of development, administration and marking.”'^ The design 
scenario throws assessment and technology people together. With the right leader- 
ship, it can generate creativity. 

Kemker corroborates. “Within the state of Florida,” she says, “the assessment 
team and the technology team, in particular, are talking more and more and realiz- 
ing how we can work together,” to create technology literacy assessments. “It has just 
opened up communications between the assessment, accountability, measurement, 
research, and technology teams. We ’re working a lot closer together,” she adds. 

Kemker does recommend a structure for organizing the efforts of all these 
creative people. Asked to discuss the design process in an lAETE-sponsored panel, 
Kemker wrote: 

The key ingredient was to have a leader that focused on the scope of the 
project, or basically the project manager. This individual understands 
every aspect of the process, so that the focus remained the development 
of a performance-based tool. At times she was very strict with the group, 
but it was definitely needed. ... It is critical to have the team leader be a 
measurement expert so that the team does not stray from the original 
scope of the project. In addition the team needs to be forward thinking 
so that the tool developed will have longevity. 
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Bazemore echoes both the appreciation of improved communications and the 
need for psychometric leadership. Recalling the experience of creating North 
Carolina’s technology literacy assessment, Bazemore says, “It has done for technol- 
ogy what the alternate assessments did with exceptional children. We have come to 
know each other. . . . We couldn’t get this job done without working 
collaboratively.” Jim Kroening with NCDPI’s Test Develop Section adds, “Those 
folks in the field who are local education agency test coordinators and others are 
now having to work with their counterparts in technology — all the way down to 
the school level — to make this happen, and that’s never happened before.” 

Asked for advice to other states, Bazemore counsels, “Always remember you 
are building a test; you cannot compromise any of the technical properties of that 
test just so that you can get it online.” She also advises making security, confidenti- 
ality of the data, and accessibility for every child early priorities. 

What Next? 

As a result of the “extreme success” of the May 2005 pilot of the Key Stage 3 
project, which included 45,000 students, Martin Ripley, director of e-Assessment 
for QCA, says, “We have proven beyond any reasonable doubt the feasibility of 
running a national test on screen — and of using the sort of approach we’ve used on 
this project.” 

That positive feedback, he adds, “has given me a platform to voice, more 
clearly, my views of where we should be going next.” He wants to bring simulation- 
based assessment to other disciplines besides ICT literacy, and he wants to start 
with science and language. “Science,” he explains, “because. . . a virtual science 
laboratory is a key tool, and it is a natural extension of the programming work. . . 
fitting behind the work that we’ve done to date. . . [and] language because I think 
that extends our capability.” Explaining that England has a required test of a scene 
within a Shakespeare play, Ripley says, “That, to date, has been a paper-based test, 
but an environment that enables us to present multimedia tasks and activities to 
students can get them to respond to more than the printed word of Shakespeare. 
You can look at production, you can look at performance issues, you can look at 
character and setting and casting issues, all on screen. That’s more of an ambition 
for me — to move into that sort of area.” Kemker, who has a background in music 
education, similarly envisions applying simulation-based assessment to music. 

Experience with assessing technology literacy has also tempted Ripley to 
encourage the development of assessments that better pinpoint a student’s decision- 
making process and understanding, or misunderstanding. Assessing problem- 
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solving abilities with “micro-world assessments” is one project he has had under way 
since 2001.'^ In one example of a task, a student is asked to grow the world’s tallest 
sunflower by controlling two variables: plant food A and plant food B. Explains 
Ripley: 

A student’s response is scored by recording the greatest height that the 
student achieved. We didn’t, in 2001, seek to define other process 
variables such as time taken by the student or eloquence of the method 
used by the student to reach the solution. Nor did we score 
optimization — i.e., the method(s) used by the student to determine that 
their sunflower could not be made to grow any taller .... These are all 
attributes that we would, if starting over with the World Class Test 
activities, now be seeking to score. 

As North Carolina’s Kroening says, “The thing that makes the online computer skills 
different from any other online test that’s out there is the fact that there are perfor- 
mance tasks. It’s not all about getting the right answer, but it’s about how did you go 
about getting to the right answer?” 
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Beyond Technology 
Competency: 

A Vision of Information 
Communication Technology 
Literacy to Prepare Students 
for the 21 st Century 



Ken Kay and Margaret Honey 

The technological and business changes brought about 
by the Web, wireless communication and distributed 
work have introduced discontinuity in where and how 
people work, how their performance is measured, and 
how their objectives are set. . . . Assignments, work 
settings, peers, employers and work choices are increas- 
ingly changeable and fluid, . . . switching from the 
employer-centric world of predefined employee activity 
and career paths to a worker-designed world in which 
individuals pursue their own portfolio of experiences, 
assignments, workplaces, projects, task forces and teams.' 

n a world where the relentless pursuit of technological progress has rewired not 
only the ever-shrinking integrated circuit but also the complex circuitry of how 
people work and businesses run, is it enough for our students to know how to turn 
on a computer and launch a program, or even how to complete a writing assignment 
using a word processing program? 
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Asking the Bigger Question 

In the last decade, technology has moved out from the periphery of our lives 
into the everyday, becoming a pervasive part of how we live, work, and learn. 
Networked communications and computer technology have transformed the 
modern workplace dramatically, touching nearly every profession and job category, 
from auto mechanic to office clerk. Skills once confined to a geeky few are now 
basic requirements for the mainstream many. As a result, the need to prepare 
students not only to participate but to excel in this technology-enriched world has 
begun to influence the thoughts and decisions of more and more teachers, parents, 
and policymakers. Such preparation is critical not only to individual students’ 
success but also to our nation’s global competitiveness. 

For our nation’s schools, this means it is no longer sufficient to target mere 
technology competency — a skills-based approach focused on hardware and software 
operations and rooted in a narrow 
definition of what it means to be 
technologically literate. The realities 
of the day demand something 
different of our schools. As lifelong 
learning and versatility^ replace 
lifetime employment and static skill 
sets, we must shed technology 
competency as the prevailing 
standard and embrace the concept 
of information communication 
technology (ICT) literacy. ICT 
literacy asks the bigger question: 

“Can students use technology to demonstrate critical learning and thinking skills?” 

Building Capacity in Our Schools 

It’s no secret that information and innovation drive the U.S. economy.^ These 
market forces also hold important implications for the U.S. education system. For 
many years, educators have emphasized the importance of technology in schools 
partly in response, albeit delayed, to the influence of technology integration in the 
workplace. 

For example, a decade after the technology boom in business, access to 
computers and other technologies has mushroomed in our nation’s schools. Accord- 
ing to the Secretarys Fourth Annual Report on Teacher Q uality, released in 2005, 
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99 percent of all schools have access to the Internet.^ By comparison, only 35 
percent of schools had access to computers in 1994, according to the National 
Center for Education Statistics (NCES). Public schools have also made consistent 
progress in expanding Internet access in instructional rooms. In 1994, 3 percent of 
public school instructional rooms had Internet access, compared with 93 percent in 
2003. And between 1998 and 2003, the student-to-connected-computer ratio went 
from 12-to-l to 4.4-to-l.^ 

Most early efforts to provide technology access operated on a computer lab 
model (a model still in use today); students would participate in one or two classes 
with a computer teacher per week.'" These classes typically focused on basic com- 
puter use and were rarely connected to what students were learning in their core 
academic classes. Recently, educators have pushed toward a more sustainable ap- 
proach to technology in education, one in which technology is integrated into 
curricula and learning. 

The steady progression of technology use and innovation in education has 
clearly had an effect on students in the United States: the majority are familiar with 
diverse technologies and capable of operating them. Nevertheless, the difference 
between operating technology and applying it to solve problems and think critically 
is a large one. Educators who use technologies in instruction commonly require 
students to develop technical skills (such as using word processing programs, 
downloading fdes, and using spreadsheets) but seldom use technology as a medium 
for developing learning skills (such as analyzing and evaluating information from a 
Web site or collaborating with others to research and fdm a documentary). It 
cannot be overstated that students will be on the outside looking in if they cannot 
use technology to find and evaluate information in a variety of formats from numer- 
ous sources and to synthesize or otherwise use that information to support ideas or 
build new conceptualizations. It is vital that we teach these skills to our students 

now. 

Responding to the New Paradigm 

Education in the 21st century will need to go beyond the notion of technology 
competency and teach students to engage with technological change in productive 
ways. Students need to know how to use technology adaptively to solve problems 
and accomplish complex tasks. 

We must embrace a new definition of what students need to know, one that 
embraces information communication technology literacy, more commonly referred 
to as ICT literacy.^ This broad concept — which acknowledges the need for students 
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to develop learning skills that enable them to think critically, analyze information, 
comprehend new ideas, communicate, collaborate, solve problems, and make 
decisions — recognizes that technology is essential to realizing these learning skills in 
today’s knowledge economy. 

Consider that today, with a $500 digital video camera and software that 
comes pre-installed on many computers, you can produce video content 
that only a few years ago would have required tens of thousands of 
dollars worth of equipment and highly trained technicians. This means 
that our communication, both personal and professional, will not be 
limited to text, and if writing is all our children are learning to do, then 
they will not be literate in the 21st century.® 

This new paradigm of ICT literacy requires a series of changes in how we envision 
the skills and tools that students and educators need. 

First, we recommend that the education community work with its partners in 
the private sector to redefine the learning skills that will make students successful 
citizens in the 21st century. Already, several factors are prompting business leaders, 
education specialists, and the general public to reexamine the skill set of U.S. 
students. Increased concern about the competitiveness of the U.S. labor force, the 
drive for more accurate assessment measures led by No Child Left Behind (NCLB), 
the growth of the high school reform movement, and the low-ranking performance 
of U.S. students on the international PISA^ (Programme for International Student 
Assessment) and T1MSS'° (Trends in International Mathematics and Science Study) 
tests are all fueling discussions about the need for a new 21st-century skill set. 
Responding to these indicators, leaders ranging from Education Secretary Margaret 
Spellings" to former Secretary of State Colin PowelP^ have signaled that todays 
students are not prepared to compete internationally. A critical first step is to define 
what skills our students need if they are to compete successfully on the global 
playing field. The Partnership for 21st Century Skills has begun to field that conver- 
sation.'^ 

Second, educators need to revise pedagogy. How teachers are integrating 
technology into the classroom needs to be wholly reexamined. By replacing the goal 
of technology competency with the goal of ICT literacy, we challenge ourselves — 
whether teachers, schools of education, or states — to see technology as a set of 
rapidly changing tools that students actively manipulate as they develop skills in 
communication, critical thinking, problem solving, and information analysis. 
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Third, the education community will need to construct and implement new 
assessment tools that measure students’ critical thinking skills and their ability to use 
information and communication technologies to demonstrate those skills. 

Going with the Flow: New Opportunities, New Skills 

Unlike previous generations, today’s students cannot rely as heavily on teach- 
ers, librarians, or publishers to act as fdters, compilers, organizers, and presenters of 
information who carefully pre-sort data, distill ideas, or select textbook and refer- 
ence books to guide students’ learning, satisfy students’ demand for information, 
and gently lead students down the path to knowledge. Today’s students encounter 
an unfiltered, unfettered environment marked by a free flow and free trade of 
information. Primary sources, statistical data sets, and mass media, once difficult to 
acquire and manipulate, now come in oversupply. An abundance of sources and 
material — of varying degrees of accuracy, usefulness, and appropriateness — are now 
available on the Web alone: by Google’s count, more than 8 billion pages. And the 
amount of data continues to grow. Worldwide we create 5 exabytes (5 billion 
gigabytes) of unique information per year, roughly 800 megabytes (the equivalent of 
a 30-foot stack of printed books) for every man, woman, and child on earth. Printed 
documents of all kinds, however, account for only 0.003% of the total.''* 

To navigate this wealth (or glut) of data, draw conclusions, and communicate 
with others proficiently, teachers and students require whole new sets of skills in 
accessing, interpreting, analyzing and evaluating complex sets of images, words, 
numbers, and sounds. Through ongoing conversations with leaders of policy, 
education, and industry, the Partnership for 21st Century Skills has identified at 
least six arenas of skills that are increasingly required in today’s workplace and 
community, presenting students with new opportunities and requiring new kinds of 
learning skills. 

1. CommunicatGEffGCtivdy: Technology’s progress has not brought an end to 
paper and pencil, but it has added to students’ communications tool kit 
audio, video, animation, design software, and other tools as well as a host of 
new environments (e.g., e-mail, Web sites, message boards, blogs, streaming 
media). Students need to master a range of digital literacy skills to operate 
these tools, which require critical intellectual awareness of how to 
communicate effectively in a 21st-century context, a context where 
communication travels across a range of tools and environments, both new 
and traditional. 
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2. AnalyZ6 and i ntarpret data: With so much data available today, students need 
to be able to look across data sets and compare, contrast, vet, and choose. 
They must be able to consider the relative merits of multiple sources and 
representations of information, because such alternatives are omnipresent. 

3. U nderstanding computational modeling: Simulations and computational 
models (such as those used to predict election results, weather systems, or 
the behaviors of solar systems) have emerged as a primary means of 
representing information and conducting research (e.g., crash testing of 
automobiles, pharmaceutical design, oil exploration). Students need to 
understand both the power and limitations of these kinds of data 
representation systems. This means not only understanding how these tools 
can support research in a wide range of scientific disciplines, but also 
understanding the mathematical, computational, and content-based 
assumptions that underlie computational models and simulations. 

4. M anaging and prioritizing tasks The increased presence of technology in 
society has necessitated a host of multitasking, selection, and prioritizing 
skills. Today, knowledge workers move fluidly among teams, assignments, 
and communities of practice within 70 percent of knowledge-based 
businesses.'^ Managing project-based work and collaborating within a 
team — skills that are expected in todays workplace — hinge on the ability to 
prioritize and complete tasks. Now more than ever, students need to develop 
and refine this ability. 

5. Engaging in probi 6m solving: In an information-based economy where 
change is constant and the value of knowledge diminishes rapidly, static 
answers to technical and intellectual problems no longer exist. Not only 
must people constantly refresh their knowledge base to stay current, but 
they must also develop the skill to apply past experiences and understand- 
ings to solve new problems, fulfill new assignments, and answer new 
questions. Students need to understand how to apply what they know to 
new situations. 

6. Ensuring SGCUrity and safety: With increased access comes increased risk, both 
to information and to people. In an environment where it takes only a few 
keystrokes to conduct financial transactions, engage in criminal behavior, or 
otherwise expose oneself to real-life risks and dangerous situations, students 
need to be armed with the insights and strategies to acknowledge, identify, 
and negotiate risk in the 21st century. 
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Figure 1 

Economy-Wide Measures of Routine and 
Non Routine Task Input (1 969-1 998) 




Expert Thinking 



Complex 

Communication 



Routine 

Cognitive 



Routine Manual 



Non Routine 
Manual 



F. Levy and R. J. Murnane. T he N 6w Division of Labor: H ow ComputersAreCreating theN ext Labor 
M arks (Princeton, NJ: Princeton University Press, 2004, page 50). (Used by permission.) 



The stakes are high. Young people who lack the ability to negotiate their way 
through the clutter face the prospect of limited success not only in school but also in 
contemporary society and in the workplace, as learning skills and technology now 
intersect nearly every endeavor people encounter (see Figure 1). Leaders should be 
alarmed that today’s students are preparing for tomorrow’s society and workplace on 
yesterday’s technology. After all, technology is not static. Technology continues to 
change, grow, and expand every day — creating opportunities for application where 
none previously existed and adding new requirements to what people need to know 
to fulfill their roles as citizens. Merely learning the technical skills to operate existing 
technologies, which are likely to be outstripped by the time students reach working 
age, falls considerably short of preparing them to succeed. 

With fields transformed sometimes overnight by technology, businesses, 
entrepreneurs, employees, and teachETSand Students alike must embrace, adapt to. 
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exploit, and anticipate technology to remain competitive; otherwise they risk 
obsolescence. Students face an economic environment in which the prospects of 
continuous, lifetime employment and a single-profession career are dim at best,'^ an 
environment in which the ability to receive ongoing training will play a pivotal role 
in their lives. In 21st-century society, only the ICT literate — those who resource- 
fully combine technical skills with an intellectual tool box enriched with experiences, 
collaboration, and knowledge — will be able to meet the challenges imposed by a 
global playing field. 

Gaining Traction: The Role of Assessments in ICT 
Literacy 

Defining ICT literacy and standards will be vital to creating learning environ- 
ments that foster 21st-century skills. However, efforts to encourage the development 
and application of ICT literacy skills will languish unless viable assessments of 
student achievement are created. The challenge of measuring 21st-century skills will 
be to develop complex, real-world tasks that capture students’ thinking. Standard- 
ized tests capture only a portion of the skills and knowledge that students need to 
know, and they do not access how students apply those skills or knowledge to 
solving problems. Therefore, these assessments are of limited value in gauging ICT 
literacy and in helping teachers 
make decisions about how to 
improve learning in the context 
of daily instruction. Instead, 

21st-century assessments should 
offer teachers ongoing diagnostic 
information that could be used 
to develop students’ learning 
over time and summative 
outcomes that quantify students’ 
progress to date. 

Measuring 21st-century skills necessitates a shift in assessment from measuring 
discrete knowledge and skills to measuring their appliGtion— the ability to think 
critically; examine problems; gather information; and make informed, reasoned 
decisions. To achieve this, we need a sustainable research and development infra- 
structure for measuring complex and real-world tasks, one that is coherent across all 
education levels. 
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Currently, Educational Testing Service (ETS) provides an 
assessment of ICT literacy, but only for higher education. The 
International Society for Technology in Education (ISTE) and 
Microsoft are developing a performance-based assessment aimed 
directly at eighth grade, the NETS Online Technology Assessment. 
Other existing assessments are designed for niche markets (such as 
economic and financial literacy, civic engagement, or global aware- 
ness) that may incorporate technology in their presentation. We need 
widely available assessment tools that are meaningful in what they 
measure and in the kind of information/feedback they offer. These 
measures need to address how students perform in complex, real- 
world tasks; diagnose students’ needs for ICT literacy intervention; 
assess the education system’s effectiveness in teaching ICT literacy 
skills; and permit students to demonstrate their proficiency in ITC 
literacy skills to educational institutions and prospective employers. 

Meeting the International Community’s 
Challenge 

Without a comprehensive plan for 21st-century assessment, 

U.S. students will continue to lag behind their international counter- 
parts in industrialized countries, who now outperform them on 
international assessments of 21st-century skills. Two-thirds of non- 
U.S. 15-year-olds performed better than U.S. students on the PISA, a 
test designed to measure how students apply reading, mathematics, 
and science content knowledge and skills in analyzing and evaluating 
information and solving problems and issues in real-life contexts. The 
PISA results show that our economic competitors recognize the 
interrelatedness of education reform, 21st-century skills, and competi- 
tiveness in the global economy and are taking action. Many countries 
have already begun the process of building 21st-century skills into 
their national education plans, while creating standards and assess- 
ment systems designed specifically to measure students’ application of 
21st-century skills. 

Britain’s Key Stage 3 (ages 12-13) ICT Literacy Assessment is an 
emerging tool that provides a good starting point for a discussion of 
what 21st-century skills assessment should resemble. Key Stage 3 



The concept of technology 
competency or profi- 
ciency became popular 
when microcomputers were 
introduced in the nation’s 
classrooms in the 1980s. 
This concept concerns 
basic skills and knowledge 
related to hardware and 
software. Early courses on 
computer skills often 
focused on basic opera- 
tions, troubleshooting, and 
programming in languages 
such as BASIC. 

21st Century Learning 
Skills, as described by the 
Partnership for 21st Century 
Skills, can be divided into 
three categories: thinking 
and problem-solving skills, 
information and communi- 
cation, and interpersonal 
and self-direction. 

ICT literacy links 21st- 
century skills to information 
communication technology 
(ICT) tools— tools that can 
change over time— to 
support the development of 
and demonstration of critical 
thinking skills and higher- 
order cognition. Examples 
given of ICT literacy, 
according to the Partnership 
for 21st Century Skills, 
include the use of ICT to 
solve problems, to demon- 
strate creative ideas or 
thoughts, to access and 
evaluate information, or to 
enhance productivity and 
personal development. 



measures content-area and thinking skills using technology and 

provides national data on students’ capabilities and disaggregated data on individual 



students. Administered online, the Key Stage 3 assessment requires students to use 
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their ICT literacy skills to solve a set of complex problems involving research, 
communication, information management, and presentation. Test activities take 
place within a “virtual town” environment that includes various visual and informa- 
tional assets (text, pictures, data, and “canned” Web sites) and a tool kit of generic 
software programs developed by the British government’s Qualifications and Cur- 
riculum Authority. These software programs provide the same capabilities as familiar 
productivity software without introducing specific platform brand bias. 

In one task, students must draft and publish a journalistic article investigating 
the ethnic diversity of a small town’s police force and teaching pool. In the course of 
their work, students use search engines; navigate Web-based information sources; 
exchange e-mails with content experts; and employ spreadsheets, word processors, 
and presentation software to analyze and present their research. They must collect 
and analyze employment data, secure source permissions via e-mail, and present 
research data in both graphic and written form. Simultaneously, the Key Stage 3’s 
assessment engine tracks and responds to the test-taker’s performance on both 
technical and problem-solving tasks by mapping their actions against the technical 
skills and learning skills they are expected to demonstrate at each level of the na- 
tional curriculum. The test engine’s final output includes both a numerical score 
(useful for national ranking purposes) and a detailed profile of the test-taker’s 
performance and areas for potential improvement. 

This particular assessment tool, while a significant step forward on the path 
toward assessing 21st-century skills and learning, is only a start. It signals an impor- 
tant trend in assessment development that the United States would be wise to follow. 
It also presents an opportunity. With few assessment tools as ambitious as the Key 
Stage 3 assessment in existence, a concerted national effort to craft ICT literacy 
standards and the means to measure students’ level of ICT literacy could position us 
to leapfrog our economic competitors and help safeguard our leadership position in 
the world economy. To help achieve this vision, we offer the following roadmap. 

Achieving iCT Literacy in the United States: 

A Roadmap 

The concept of ICT literacy acknowledges technology as a constantly evolving 
and ubiquitous tool that can be used to accomplish a multitude of tasks. Teaching 
ICT literacy skills effectively in schools requires that we alter how we think about 
learning and technology, migrating our focus from technology competency to ICT 
literacy. This transition hinges on a comprehensive reexamination of four major 
areas: public policy, standards, assessment, and research and development. Key 
questions and suggestions in each area follow. 
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Public policy. Public policy will set the tone for the transition from technology 
competency to ICT literacy in our schools. Building consensus among lawmakers 
and other stakeholders is critical to implementing policies that support ICT literacy 
standards, assessment, professional development, and research and development. 

First, citizens and leaders alike must recognize that effective education policy in 
the 21st-century needs to acknowledge the real-world transition from school to life 
and work by focusing on ICT literacy. Opening up a dialogue between the private 
and public sectors will help strengthen the idea that students are in a “pipeline” that 
carries them from school into the workplace and society at large. ICT-literate 
students will be able to think critically, solve problems, and communicate effec- 
tively — skills that will be vitally important to their success in and out of school. 

The federal government also plays a critical role in public policy, and it is 
currently in an important position to change technology learning standards. While 
the No Child Left Behind Act of 2001 (NCLB) established an eighth-grade technol- 
ogy literacy requirement, the requirement is not a full statement of ICT literacy and 
does not include a mechanism for ensuring accountability. 

What steps can betaken to ensure that ICT literacy is recognized and supported by 
state and federal policy? By taking the lead in revising the eighth-grade technology 
literacy requirement, adopting a national standard consistent with ICT literacy, and 
establishing clear guidelines for assessing states’ progress in meeting the standard, the 
U.S. Department of Education could move ICT literacy forward. Such revisions 
could be included in the reauthorization of NCLB in 2007. When this reauthoriza- 
tion comes up, the education community, as well as state and federal departments of 
education, can urge Congress to modify the legislation to include the complete ICT 
standard and accountability mechanism. Furthermore, educators and policymakers 
at all levels can establish mechanisms that hold schools accountable to the ICT 
literacy standards by providing them with the tools and skills to teach and assess ICT 
literacy effectively. 

Standards, it is important to student success that every state adopt and embed 
into core curricula a set of standards for ICT literacy by 2010. These standards 
should be comprehensive and crosscutting. They should also transcend the speed at 
which technology changes by focusing on skills that would apply to any technologi- 
cal tool. For example, one of the ICT literacy standards developed by the Partner- 
ship for 21st Century Skills is using technology to communicate effectively. This 
standard would apply to current technologies, such as presentation software or 
handheld devices such as portable digital assistants (PDAs), as well as to future 
technologies. 
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While ICT literacy standards should emphasize the skills that technology 
facilitates, and not the technology itself, it is critical to recognize that the impl6m6n- 
tationSof the standards are not static. As technology evolves, educators will need to 
revisit the way that ICT standards are implemented. The business community will 
be an important partner in that business leaders can help educators understand how 
technology is changing, and this knowledge can help educators implement ICT 
literacy in schools. 

How can states embark on the task of generating new standards for ICT literacy? 

W ho should be involved? First, organizations in a position to define ICT literacy, such 
as the State Educational Technology Directors Association (SETDA), the Interna- 
tional Society for Technology in Education (ISTE), and the Partnership for 21st 
Century Skills, need to meet and agree on standards for ICT literacy that feature the 
use of technology to acquire and demonstrate learning and learning skills. While 
developing these standards may sound daunting, the skills described by these 
standards will be content independent, meaning they will apply across content areas 
and can be addressed at every grade level. 

Second, state education leaders and technology directors are encouraged to 
examine efforts to create assessments of ICT literacy in the United States and the 
United Kingdom to determine the degree to which current and emerging online 
assessment tools can be aligned with the standards. 

Third, state education leaders and technology directors need to develop a 
strategy for periodically consulting the broader business community for advice on 
the specific technologies most closely associated with the learning skills identified in 
the standards. This relationship between business leaders and educators recognizes 
that while skills encapsulated in ICT literacy standards are likely to remain relatively 
constant, the technologies involved will likely change every few years. For example, 
if students need to demonstrate their ability to effectively communicate in a 21st- 
century context, their performance should be demonstrated by using industry- 
standard tools to address realistic and relevant issues. While the principles of good 
communication, as embodied in the standards, aren’t likely to change, the modes of 
effective communication, as identified by the business community, will change over 
time. 
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Comparison of Skills Across Approaches 



Tech Competency 
Examples 


NETS-S Standards'® 


ETS ICT Literacy 
Skills^" 


21 St Century Skills 
ICT Literacy ModeP' 


This commonly used 
approach is based on 
operating different tools and 
emphasizes the following: 

• Basic computer 
operating skills 

• Use of technology for 
rote memorization 

• Repetitive use of 
software without 
adaptation to task 

• Use of technology to 
gather undifferentiated 
information 


Student technology 
foundation skills are 
organized under six broad 
skill categories: 

• Basic operations and 
concepts 

• Social, ethical, and 
human issues 

• Technology productivity 
tools 

• Technology communica- 
tions tools 

• Technology research 
tools 

• Technology problem- 
solving and decision- 
making tools 


Comprehensive test of ICT 
proficiency for the higher 
education environment. 
Offered at two levels of 
difficulty; test-takers use 
technology to demonstrate 
seven proficiencies: 

• Define 

• Access 

• Manage 

• Integrate 

• Evaluate 

• Create 

• Communicate 


Emphasizes the ability of a 
student to use technology to 
demonstrate the following: 

• Critical thinking 

• Problemsolving 

• Communication 

• Collaboration 

• Project management 

• Global awareness 

• Civic engagement 

• Business and financial 
literacy 



Assessment. Although the federal government has emphasized assessments for 
the purpose of accountability as demonstrated by the need for schools to measure 
and report adequate yearly progress (AYP), current assessments do not measure the 
skills our students need for the 21st century. In response, the Partnership for 21st 
Century Skills has worked to build consensus around ICT literacy and has articu- 
lated a plan for 21st-century assessments: 

The creation of diagnostic, formative, and summative assessments that 
measure students’ content knowledge, skill development, and ICT 
literacy present an important opportunity to feed a variety of assessment 
data back into our nation’s school systems at their appropriate levels, 
from the trend and pattern data needed by policy makers, state officials, 
and district administrators to the individual and class level student data 
on particular tasks, skills, and topics needed by classroom teachers, 
students, and parents. While creating more diagnostic, performance- 
based measures requires some investment, embracing the teaching and 
assessment of 21st century skills presents us with a unique opportunity to 
leap ahead of our competition in the global community. The Partnership 
[for 21st Century Skills] believes that the movement to embrace 21st 
century skills will be greatly enhanced by identifying ways to measure 
these skills. Conversely, the lack of such assessments will hinder progress 
toward widespread adoption of 21st century skills. 
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As the Partnership for 21st Century Skills indicates, it is critical that the nation’s 
investment in assessment development, which is expected to reach several billion 
dollars^^ between fiscal years 2002 and 2008, focus not only on fulfilling the federal 
mandate, but also on addressing our children’s ICT literacy skills. The private sector 
has a critical role to play in developing ICT literacy assessments. The for-profit 
assessment community’s experience in assessment design makes it an important 
partner for educators who need to develop effective ways to assess ICT literacy — an 
indicator of children’s preparedness to meet the demands of tomorrow’s workplace 
and global community. 

How can states address the need for ICT literacy assessments? Can the private sector 
be more involved in developingICT literacy assessments? Bridges must be built between 
the educators who will administer assessments, the policymakers who want reports on 
assessments, and the private sector that will be called on to develop those assessments. 
Private sector companies are beginning to measure ICT literacy within higher educa- 
tion. States could capitalize on these efforts and create testbeds for these tools in the 
K-12 environment. It is important that state education leaders examine the pioneer- 
ing investments being made by the United Kingdom in ICT literacy assessment and 
draw on these ideas to create similar measures here. Another possibility would be 
meeting with private sector companies in the United States to better understand their 
capabilities in measuring ICT literacy and offering to help them create pilot projects 
that foster the creation of ICT literacy assessments. In conjunction with these efforts 
to foster connections between assessment developers and schools, a robust dissemina- 
tion mechanism needs to be established that allows states to tap into the results of 
these pilot collaborations. The Institute for the Advancement of Emerging Technolo- 
gies in Education (lAETE) at AEL has taken some initial steps through its annual 
conference series on technology-supported assessments (www.edvantia.org/publica- 
tions/indexl.cfm?&section=publications&area=publications&id=631). The Partner- 
ship for 21st Century Skills has also launched an online database. Assess 21 
(www.21stcenturyskills.org/assess21/) for tracking the creation of ICT literacy 
assessments worldwide across key 21st-century dimensions. 

Research and development. Building an education system that effectively 
teaches ICT literacy will require a national research and development infrastructure 
for developing standards for teaching as well as assessments of learning cognitively 
complex and real-world-related tasks. Currently, there is not a sufficient foundation of 
research and development to support the full range of tools that could be used to 
teach and assess ICT literacy in the 21st century. 

The federal government could assume an important leadership role in establish- 
ing this infrastructure to ensure high-quality, cutting-edge research. Other govern- 
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ments around the world have already begun to make these investments. A commit- 
ment to funding research on ICT literacy and 21st-century skills will help to 
facilitate a more uniform approach to teaching and measuring ICT literacy nation- 
wide. The federal government could also work with state and local leaders to develop 
ICT literacy tools and assessments and to build the infrastructure and action agenda 
necessary for implementation. Education research institutions, both in higher 
education and in the public sector, can also play a role in the production of research 
and development for ICT literacy. 

H ow do we foster a national research agenda in ICT literacy? National and state 
education leaders and the business community need to make a clear and consistent 
case for ICT literacy’s critical role in our educational systems, present and future — a 
case that commands a higher priority in our nation’s schools, schools of education, 
and communities. Increasing federal funding will be needed to meet the challenge 
head-on and to create real momentum for change. At the same time, education 
agencies need to cooperate broadly and specifically to develop strategies that maxi- 
mize and extend existing resources in teaching ICT literacy. 

201 5: An Education Odyssey 

Shifting the focus in education from technology competency to ICT literacy is 
a daunting yet necessary task. However, embracing ICT literacy as a goal worthy of 
pursuit will require philosophical changes in terms of how we prepare children in 
schools, financial changes related to public policy and research and development, 
and organizational changes related to the way schools are structured for 21st-century 
learning. These changes are well worth the investment of stakeholders who are 
committed to preparing the nation and its students for success in the 21st century. 
Twenty years from now, we may look back on the term tSihnology Competency the 
way we look back today at rote memorization — as a lower-level skill that simply does 
not prepare young people for today’s technology-driven, global society. With a well- 
coordinated roadmap and a vision of what a new culture of learning reinforced by 
21st-century skills and assessments will offer, ICT literacy will be an achievable goal. 
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